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ABSTRACT 

This guide aims to help rehabilitation facilities 
develop effective self -evaluation systems to improve quality 
assurance for supported employment. The guide proposes chat quality 
be measured by emphasizing direct outcomes that meet consumers' 
specified needs and concerns in terms of decision making, 
determination of quality, and access to choice. A section on 
approaches to quality assurance discusses levels of quality 
assurance, national regulations, national standards, and quality 
through the promotion of exemplary practices. A rationale is offered 
for voluntary monitoring of quality assurance. Evaluation criteria of 
model monitoring systems are specified, and steps involved are 
outlined. A modified system is then proposed by the National 
Association of Rehabilitation Facilities (NARF). The system is based 
on management practices that promote quality, management components 
of exemplary supported employment programs, and categories for 
evaluating quality of exemplary programs. The NARF system includes 
two basic components: (1) the NARF Quality Indicators Profile which 
functions as a self -checklist ; and (2) the Six Guiding Principles of 
outcomes, vision, consumer satisfaction, collaboration, 
organizational development/staff development, and self -evaluation. 
(42 references) (JDD) 
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For^vord 



This monograph, Thm Role of Voluntary Self-Assearritent in Quality Assurance, is one 
of thrift managaiMnt monographs which NARF has developed to meet the needs of 
executive directors and program level managers who are developing and expanding 
their supported employment services. The other two management monographs are: 
Effmctivm Managmment of Supported Employment and Consumer Choice and Satisfaction* 
All three publications combine information on the latest trends with concrete 
suggestions for action. 

NARF wishes to thank the many people who .assisted with reviewing and revising these 
monographs, including NARF's National Scope Supported Employment Advisory Council, 
NARF's Supported Employment Task Force, and NARF headquarters staff. Special thanks 
to Oianne Greyerbiehl, Chip Beziat, Fred Mens, Richard Culp-Kobinson, ?eggy Todd, 
Alan Goldstein, David Price, Janet Samuelson, and Terry Edelstein for their feedback 
and assistance in developing this final version. And finally, thanks to the taany 
rehabilitation providers who have made tremendous strides \/ith supported employment. 
Your success not only provided a foundation for our national efforts to disseminate 
best practices information, but more importantly, your success has enriched the 
lives of many consumers. 
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A Climate of Change 



Today, the use of supported employment is continuing to expand and rehabilitation 
providers are implementing a variety of procedures to assure its endurance as a 
rehabilitation option. Supported em.^loyment today, however, is being shaped and 
molded by several major concerns and activities* These include: an emphasis on 
serving the most severely handicapped (Wehman, Kregel, & Shafer; 1989); multiple 
concerns regarding access to and assurance of long-term funding; the delivery of 
support services by a variety of providers, including state MR/DD agencies; the 
expansion of employer-sponsored supports; and the revision of supported employment 
regulations (Federal Register, 34 CFR 363, February 13, 1990). Additionally, the 
Americans with Disabilities Act — the civil rights bill for people with disabilities- 
-is close to passage, and preparations are being made for the reauthorization of the 
Rehabilitation Act which is scheduled for 1991* 

Accompanying the Americans with Disabilities Act are many activities by consumers 
tnat have already impacted the way the rehabilitation field conducts its business. 
For example, in January, 1989, consumers participated in a conference on self- 
determination sponsored by the Office of Special Education and Rehabilitative 
Services. One outcome of the conference was a monograph with 29+ recommendations 
(Perske, 1989) for greater concumer input and involvement in the rehabilitation 
field. Recommendations from that conference included: consumer involvement on grant 
peer review panels, consumer involvement in governr:siit policymaking, and the 
development of model programs exemplifying self-determination attitudes and 
practices. These are already being implemented. 

The National Association of Developmental Disabilities Councils also has completed 
a review of 13,000 interviews with consumers — the largest sorvey of this group to 
date (JasKulski, Metzler, & Zierman, 1990). That report found that supports are 
needed to overcome barriers to independence, productivity and integration and that 
funding mechanisms are needed to support individual choice and control, including 
funding individuals rather than facilities. The essence of the report is that 
people with •'developmental disabilities can live in the community; many need 
supports to assiJt them . . . Supports to individuals will enable choice and 
selection, help people to stay in their homes and communities, and empower people 
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to obtain what they need to achieve the goals of independence, productivity and 
integration." (National Association of Developmental Disabilities Councils, 1990, 
p. 4). 



The SystMatic Phase 

In the United States, supported employment for the most part currently is in the 
phaae of implementation which Schalock (1988) refers to as the second stage, the 
"systematic phase" which focuses on systems of service delivery, measurability, and 
reportability. The systematic phase is one of monitoring and compliance review. 
Implementers and policy makers are interested in consistency and quality, and are 
particularly interested in how to assure that certain standards are maintained, with 
outcomes achieved and costs contained. At the same time, implementers are concerned 
about the modification or elimination of rules and guidelines which provided some 
protections, even as they may have hindered implementation (Federal Register, 34 CFR 
363, August 14, 1987). So we are entering an age of demands for flexibility and 
at the same time, an era of quality expectations • 

Lessops froH PL 94-142 

Implementers and policy makers have had the luxury of watching the implementation 
of PL 94-142, the Education for All Handicapped Children Act of 1975, and of 
observing and participating in decisions regarding the reauthorization cf that 
legislation. That background has provided an impetus to avoid some of the pitfalls 
such as early adoption of specific strategies — such as the resource room — with later 
needs to reevaluate strategies and redesign approaches (in the case of the resource 
room — to achieve integration). The rehabilitation field also has had access to the 
data on PL 94-142 which indicate that physical integration alone is not effective. 
Thus, researchers (Calkins & Walker, 1990; Chadsey-Rusch, Gonzales, Tines, & 
Johnson, 1989; Rusch, Johnson, & Hughes, 1990; Storey & Knutson, 1989) have become 
involved in designing systems and approaches to measure and promote social 
integration with the desired interaction with coworkers and supervisors. 

PL 94-142 has provided an opportunity to observe the effectiveness of legislation 
that has provisions for monitoring adherence through "individualized educational 
plans," with due process provisions for recourse if individuals are denied an 
appropriate education. While teachers, the implementers of the policy, have 
expressed concern over the paperwork involved, many parents and children have been 
afforded due process and access to appropriate public education through this 
entitlement legislation. 
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While the adult services rehabilitation legislation is not generally viewed as 
entitlement legislation, many of the dilemmas faced by education also have been 
concerns of the adult rehabilitation community. These include: access to services 
and procedures for eliminating waiting lists, provisions for assurance of the most 
appropriate training, and the desire for integration. Additionally, the supported 
employment program in particular has empnasized outcomes in terms of wages, hours 
worked, integration, benefits, and level of disability served, with a priority 
emphasis on those persons with severe disabilities and chronic unemployment. 

Proaoting Quality 

National discussions of quality assurance (Isbister, 1989) have led to agreement to 
promote quality through the identification and subsequent dissemination of exemplary 
practices (National Association of Rehabilitation Facilities (NARF) , 1989b). 
Several task forces working with the National Supported Employment Panel of Experts 
have been created to focus on the exemplary practices in the following &reas: family 
involvement, consumer involvement, management competence, and specific component 
practices such as the creation of career ladders, social integration, and designs 
across disabilities. 

Programs are being developed to assure quality through the implementation of program 
standards and certification programs. The Commission on the Accreditation of 
Rehabilitation Facilities (CARF), for example, last year developed a new section of 
standards specifically on supported employment (CARF, 1990) and various universities 
are designing two-year, four-year, and master's degree certification programs. 

Quality asrurance in supported employment is being discussed. Publications are 
being disseminated which urge, promote, and demand the involvement of people with 
disabilities in the empowered arena of decision making (Perske, 1989). Hncent 
presentations ' rehabilitation providers and pjlicy makers have brought to the 
forefront thu . suits of over 13,000 interviews of individuals with disabilities 
conducted by state developmental disabilities councils (McFadden, 1990) > These 
interviews describe people with disabilities as being unemployed and underemployed, 
lonely, and dissatisfied with the overly protected and restricted lives they have 
lived under the programs that have been implemented over the past twenty years. 
These reports have resuitecf In over 3,000 recotmeiid^tiona for Improved services* 
Testimony by people with disabilities, such as Mary Jane Owon with Disability Focus 
Incorporated (Owen, 1990), includes the demand to "suffer the consequences of 
reckless behavior and rove away from the protection from the harsh realities of life 
that everyone else lives." 
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Dufresne (1990) reports that many people with disabilities: 

o live with people they did not choose to live with, all of whom have 
disabilities. 

o work at places they did not choose to work with co-workers with disabilities, 
o participate in the community through segregated recreational a*.tivities which 
are not of their own choosing. 

Dufresne (1990) is concerned about the way the word "choice** has been used when 
describing the choices of people with disabilities and has provided an extended 
explanation of choice which includes: "option(s] (implying a power to choose that 
is specifically granted or guaranteed); alternative[s] (implying a necessity to 
choose one and reject another of possibilities); selection (implying a wide range 
of choice); and election (implying an end or purpose which requires exercise of 
judgment) . " (p.i7) . 

Similarly, CARF (1990) has defined informed choice as: "decipions made by a person 
served which are based n sufficient experience and knowledge, including exposure, 
awareness, interactions, or instructional opportunities, so that the choice is made 
with adequate awareness of the alternatives and the sequences of the options 
avail«.ble. ** 

While the rehabilitation field is beginning to define more adequately the concepts 
of choice and quality, a clear message from recent discussions with consumers is 
that standards that h^ve been used to date are woefully inadequate (Roberts, 1989; 
Mitchell, 1988; Schwier, 1990). The question faced by policy makers is "what 
standards should be developed and implemented?" or "what is needed to actually 
assure quality?" 

What "Quality" Should be Assured? 

Information from consumers suggests that quality should be measured first of all by 
an emphasis on direct outcomes: outcomes that meet consumer's specified needs and 
concerns. The needs and concerns which recently have been voiced include the need 
for: 

1. Consumer input into decision making. 

2. Consumer determination of quality (Goode, 1990). 

3. Outcomes which give consumers access to choice in terms of: 

where they work and the work they do 
where they live 
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with whom they work 
with whom they live 

who aesista with their daily personal care needs 
what they do in the community 
how they spend their time 
risks they decide to take 
support services used 

the income and accompanying work conditions they choose to pursue, and 
how they spend their money* 

Others also have addressed outcome measures (Schalock, 196>8; Bellamy, Rhodes, Mank, 
C Albin, 1988; Sandow, Rhodes, Mank, Ramsing, & Lynch; 1990). Programs desiring 
more information should have little difficulty accessing opinion on outcome 
criteria* 

Approaches to Quality Assurance 

Levels of Quality Assurance 

Quality assurance can be attempted through formal activities at various levels 
within the service delivery system, including measuring the impact on the 
individual, the implementation by the local provider and community, the provisions 
established at the state level, and regulations and standards that are national in 
scope. Informal activities at each of these levels can also facilitate assurance 
of quality. 

National Regulations 

The supported employment regulations implemented in August, 1987 recently were 
opened for review (Federal Register, 34 CFR 363, February 13, 1990). The federal 
government has expressed a need to examine specifically the value of: (1) the 
twenty hours per week work requirement; (2) availability of post-transition or post- 
closure services from the VR state agency; (3) definitions of "ongoing support 
services" and "extended services;** (4) clarification of the exemption of the 
chronically mentally ill from the job skill training services requirement; and (5) 
clarification of job skill training. 

The Consortium for Citizens with Disabilities Employment Task Force, a Washington, 
D.C.-b&sed consortium of representatives from national professional and advocacy 
organizations and disability groups, recently held a series of meetings, reached 
consensus, and forwarded comments to the Rehabilitation Services Administration 
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regarding supported employment regulations. Among their recommendations were 
several strengthening the use of the IWRP as a criterion for determining the most 
appropriate services. For ex£unple, in regard to the "twenty hour" requirement the 
group recommended replacing the requirement with the phrase: "as defined in the 
IWRP." The group also recommended replacing the existing language exempting persons 
with mental illnoss from the job skill training requirement with: "Support services, 
or assessment must occur with client contact at least twice monthly, on or off the 
job site. The nature and frequency of the onsite assessments and the job skills 
training is determined by the IWRP. Transitional employment may be considered an 
acceptable outcome for employability . " 

The IWRP recommendations represent examples of national regulations which could 
strengthen the role of "individualizing" approaches to supported employment and 
hence, could effectively implement quality through changes at the national, state, 
and local systems levels. At the same time, enforcement of proposed regulations 
could require additional resources. The current regulation of "twenty hours" of 
employment from the first day of supported employment does not permit its use with 
many individuals for whom supported employment was originally developed — persons 
with the most severe disabilities. However, the proposed reference to the IWRP 
implies that a system will be established to monitor the appropriateness and 
effectiveness of implementation. Such a system could require additional paper 
compliance. Sxnce the IWRP ij already a requirement, some would argue that the 
burden would not be the addition of new monitoring procedures, but rather one of 
more effeucive rncnitoring within the framework of existing requirements. 

National Standards 

The Commission on Accreditation of Rehabilitation Facilities (CARF) , an organization 
that uses rehabilitation providers to survey and access adherence to organizational 
standards, represents one level of self-monitoring. CARF is moving in the direction 
of quality assurance through a focus on outcomes rather than process. The new 
Supported Employment Standards, for example, attest to the focus on outcomes. This 
is evident through the inclusion of such items as those listed in Table 1. 
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TABLE .1 

Examples of CARF Outcoaa-R«f«r«nc«d Stat«n«n^s 



The program evaluation system in a supported employment program should address a 
variety of measures, some of which should be average number of weeks worked, average 
number of hours worked per week, earnings and benefits, job retention, job 
advancement, and job changes. 



Each person should have an individual writte.. plan that, at a minimum, includes wage 
range for the job, best job match, optimum integration and independence, career 
options, length of time from referral to placement, integration, number of 
successful placements classified by severity of disability. 



The Accreditation Council on Services for People with Developmental Disabilities 
(ACDD) also has established standards for implementation of supported employment, 
several of which address integration (ACDD, 1987, p. 15). Gardner and Parsons (1S90) 
describe the ACDD accreditation process as one which represents a "synthesis of 
emerging values and technologies with traditional norms and practices. The dynamic 
integration provides ein evolving consensus model of quality." (p. 207). The 1987 
standards include 47 standards regrading protection of rights of individuals and 23 
standards covering normalization, age-appropriateness and least restriction. The 
1990 standards contain five standards on consumer empowerment, 28 standards on 
community integration and socialization, 51 standards on protection of individual 
rights, and 24 standards on normalization^ age-appropriateness and least 
restriction. While the emphasis appears to be in the right place, the sheer number 
of standards may hinder effective implementation as agencies attend to details 
rather than the overall life of the individual* Although specific protections 
against abuse are needed, a holistic individual perspective also is a prerequisite 
to quality. 

Quality through the Promotion of Exemplary Practices 

Rather than use standards ov evaluation systems as the primary guise for expansion 
of quality practices and determent of ineffective ones, some arguments prevail for 
the primary emphasis on voluntary implementation of quality through the 
dissemination of information regarding quality practices (NARF, 1989). In the 
business world, such an approach is the everyday reality of the marketplace where 
supply and demand are the factors that influence not only costs and prices, but also 
quality. Tom Peters, author of Thriving on Chaos (1988) has stated: 
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Truly decentralized, externally (customer-) obsessed units, with a clear 
vision and high involvement, are more under control in today's volatile 
environment than traditional, centrally controlled units — which are 
inflexible and more out of touch by definition • • • the former emphasize 
market-driven decision-making and fast adaptation* (p. 476)* 

Attendance at seminars stressing quality and demand for publications addressing 
quality practices in supported employment is currently high and predicted to remain 
so for the next few years (NARF, 1989b). NARF*s experience is that rehatilitation 
providers are interested in how to be cost effective , how to deliver quality 
services, and how to obtain quality results for consumers* The question NARF 
continues to ask is, "how does dissemination of information on quality impact 
programs achievino poor results?" While NARF has not identified a set of providers 
achieving poor results with supported employment, NARF does have information on the 
larger grcup of providers in general* 

Evaluation reviews from Beminars and products on exemplary practices indicate that 
programs are differentially impacted by information on quality services, with 
significant changes sometimes occurring* For example, after participation in a NARF 
retreat on the future of supported employment, several executive directors reported 
an acceleration in the rate of supported employment placements* 

As with other systems change efforts (Hall & Loucks, 1977), implementation will 
occur in various stages, with early adopters, the later mass of adopters who become 
involved, and finally, the late adopters* Several research studies have verified the 
effectiveness of concentrating systems change efforts primarily on the early and 
middle adopters* With this approach many of che late adopters are impacted by the 
influence of others and the final necessity to change* As with the late adopters, 
ineffective programs achieving poor results are likely to be impacted by changes 
occurring around them* Thus, raising the overall quality level through a focus on 
excellence and dissemination of information on excellence should an effective 
strategy for increasing competence for programs achieving less than satisfactory 
results* Of course, the field also may benefit from other efforts targeted to this 
latter group* 

Voluntary Monitoring 

Wi**ck (1990) and Sandow et al* (1990) have presented some convincing arguments for 
the use of voluntary measures in assuring the quality of supported employment. A 
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major component of wieck*8 argument is that alternative approaches (accreditation 
ptandards, regulations) are neither cost effective, efficient » or effective. Wieck 
suggests that often paper compliance occurs without the requisite increase in 
quality. Sandow e: al. (1990) have presented a similar argument, including the 
problems perpetuated by focusing on "procesb** rather than "outcome** standards. 
E.Tisting standards (ACDD, 1988; CARF, 1990) have detailed procedures that should be 
followed in order to achieve accreditation. However, the reports of the minimal 
outcomes achieved overall by persons with disabilities (McFadden, 1990) point to the 
discrepancies between adhering to process guidelines and actually placing a greater 
priority on outcomes such as wages, hours worked, integration, personal choice, and 
sel f -det ermina t ion . 

While Wieck (''«>90) and Sandow et al. (1990) havt advocated for voluntary standards, 
they have chosen alternative avenues to further the quality of supported employment. 
Wieck (1990), as Executive Director of Minr4esota'3 Governor's Planning Council and 
with a history of involvement in and advocacy for people with disabilities, has 
urged the use of monitoring by community volunteers who evaluate the **humaneness of 
the environment ** Such a system has been used to evaluate residential programs and 
has resulted in the evolving use of practices that enhance the independence of 
people with disabilities. 

Minnesota's voluntary monitoring system usi'S trained volunteers with an affiliation 
agreement wi^h rehabilitation service providers. Volunteers receive at least eight 
hours of training and focus their reviews on quality services, normalization, 
quality of life, and abuse/neglect concerns. Volunteers are trained to examine 
quality of life in sir areas: environment, rights, use of community resources, 
personal relationships, staff involvement, and commitment to personal growth. Vhe 
Minnesota Governor's Planning Council on Developmental Disabilities ( ARC-Minnesota, 
undated) suggests that the following guidelines be used: (See Figure 1). 



Age-Appropriateness 

Activities/tasks would be appropriate for non-disabled peers. 

Staff model appropriate adult behaviors (address people as aoults, use age 

appropriate reinf orcers) . 

Decorations and materials are appropriate for non-disabled peers. 
Schedule and routine are bas'' ^ on schedule of adults who are not disabled. 



cont. on next page 
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Productivity 



* Activities are meaningful and functional 

* Number of hours worked by week/roonth/year 

* Wages (hourly/piece rate) 

* ChangeB in wages over time 

* Length of time on job 

* Income covers his/her living needs 

Independence 

* services are as least intrusive as possible 

* Reduct.L n for need for servicas over time 

* Reduction in cost of specialized support or training 

* Activities lead to personal growth, development, and personal satisfaction 

* A means of communication exists to allow daily interaction with primary 
people (speech, signing, adaptive devices) 

* A means of mobility exists to move about home and community environments. 

Functional Activities 

* Tasks and activities are relevant to daily life and use real materials. 

* People are taught how to spend their money, how to prepare food, clean house, 
shop and other skills to live on their ^^wn. 

Integration 

* Amount of time spent in integrated settings. 

* Use of generic resources (transportation, parks, recreation. Adult Education, 
library) 

* Number of interactions with non-handicapped peers — there are opportunities 
to have interactions with non-handicapped peers. 

* Number of people with disabilities is less than 3% of total people in a 
setting (such as a work force). 

* Opportunities f:r friendships with non-paid, non-disabled peers. 

* Support occurs in heterogenous groupings 

Learning in Natural Environments 

* Skills must be taught in a variety of environments because Df limited ability 
to generalize from one environment to the next. 

* Activities and training occur in natural environments (at a minimum in 
community living, supported employment, and recreation/ leisure. ) 

Choice and Decision Making 

* Participation in decisions chout use of personal income 

* Participation in decisions about home, choice of location, furnishings, and 
decor 

* Lifestyle choices encourage wellness - nutrition, weight, smoking, strest. 
relief, emotional support, and appearance. 

Figure 1. Minnesota Oovernor*s Planning Council on Developmental Disabilities* 

Guidelines for Evaluating Quality of Life 
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In contrast to the use of volunteers in Minnesota, the system proposed by Sandow et 
al. (1990) uses agency staff for self-monitoring. Like NARF, Sandow et al. have 
borrowed from quality assurance in the business field to develop a system for 
implementation. The system proposed by Sandow does not use a prescribed instrument 
for assessing quality, rather a process is advocated. The steps are presented in 
Figure 2. 



1. Expand the involvement of the local community. 

2. Establish a clear mission or purpose. 

3. Identify aud stratify the accomplishments of the mission. 

4. Stratify accomplishments into key processe'^. 

5. Define measures. 

6. Create useful information. 

7. Use the information to take action. 

Figure 2. Steps proposed by Sandow et al. for Agency Self-Monitoring 

The system proposed by San^ov: et al. starts with community involvement and moves to 
redefinition of purpose and ongoing monitoring of the organization's accomplishment 
of that mission. A critical result of identifying and stratifying the 
accomplishments of the mission according to Sandow et al. is the development of a 
trouble-shooting tree, a tree which includes the essential requirements of 
supported employment (paid work in the community, etc) . The tree provides a backdrop 
for agency problem identification and resolution. 

The next step. Step 4, involves stratifying accomplishments into key processes. For 
supported employment this could include such activities as identifying prospective 
jobs an:l training workers to perform work — once again providing a basis for problem 
identification and resolution. 

Step 5, defining measures, includes a task analysis of the steps needed to complete 
the tasks. For paid work to be available, for example, individual wages, number of 
hours of paid work and consumer match with job are needed. The final steps involve 
creating useful information — or translating the ^.ata into the information needed for 
decision making — and then using the data to take needed actions, comparing past and 
current performance. 

The approach taken by Sandow et al., as explained in their article in the Journal 
of Rehabilitation Administration, involves translating a theoretical approach to 
problem solving steps for the identification of needed action. Discussions with 
staff at the University of Oregon indicate that participating in training seminars 
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on this process may facilitate understanding and adoption of their model for self- 
monitoring. 

While the system developed by Sandow (t al . should be of assistance to providers, 
NARF is concerned that some agencieb . .y become bogged down by the process of 
translating component activities to unique situational circumstances, NARF 
therefore he^s developed an option that we believe is more straightforward and may 
contain fewer risks in tertrs of possible misinterpretation — a system which many 
providers should be able to implement after reading this monograph. If providers 
want additional information or training, they are encouraged to contact NAKF for 
assistance in implementation. 

An additional difference between the NARF and Sandow et al. approach is that the 
NARF model focuses on steps rehabilitation providers can achieve whether or not they 
initially choose to expand involvement in the local community, certainly supported 
employment necessitates greater community involvement; however, agencies may wish 
to begin with an initial self-evaluatiun before making any changes, including 
increased com.'nunity involvement. 

NARF's Position 

As the national trade association representing vocational and medical rehabilitation 
facilities, with over 750 members and 28 state chapters, NARF is most interested in 
promoting quality through "self-monitoring" and thus we are supportive of the 
concept articulated by Sandow et al. Quality should start not with exteriidi 
"policing" bat rather by internal self -management and self-control. 

To summarize NARF's perspe'-tive regarding the differential systems of quality 
assurance as implemented through dissemination of information on exemplary practices 
as well IS the use of both voluntary and regulatory standards, NARF is supportive 
of concurrent developments across all areas. NARF, however, is concerned about the 
potential drain on resources through various demands on systems. The emphasis on 
quality through process standards in particular sometimes has led to standards which 
contradict individual needs (Quality Assurance, 1990; NARF, 1988). Particularly 
vivid examples include program standards that regulate time, meals, and privacy 
while ignoring individual wants, desires, or needL. NARF urges the careful 
evaluation of all such regulatory standards and the appropriate revisions according 
to a criteria respecting individual wants and desires, with the principles of self- 
determination and choice receiving priority status. 
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NARF is aupportive of the concepts of voluntary monitoring as developed by Wieck, 
proposed by Sandow et al., and implemented by CARF. NARF supports all three 
approachefi and believes each has validity, particularly when used in tandem. The 
cc tple^l^y of both the CARF standards and the system recommended by Sandow et al. 
f-Are, however, of particular concern to NARF, 

NARF* 8 Proposal: The Basis 

Both the CARF system and the Sandow suggestions include critical elements that may 
hinder their direct and immediate usefulness in establishing a climate of quality; 
i.e., the complexity of the approach may interfere with the intent of the program. 
Hence, NARF is proposing another meanp for self-monitoring — a simple way of using 
self -assessment to enhance quality, NARF*s proposal is based upon an analysis of 
routes to quality and on management theory, most specifically that of Tom Peters 
(1989) as elaborated in Thriving on chaos and Goodrick (1985) as developed for state 
mental health systems. This theory says that quality should be promoted through 
attention to the following (See Table 2): 



TABLE 2 

tanagement Practices that promote Quality (froB business management practices, 

Ooodrick, 1985; Peters, 1989) 

Vision of top management, clearly articulated 
Intense, purposeful listening 
Customer satisfaction 
Constant self evaluation 

Being a step ahead of competitors (rapid change) 
Flexibility 

Field testing (finding small errors quickly) 
Fostering collaboration 

Visible, decentralized management that sets an example 

Functional working groups 

Enhanced results and greater productivity. 



NARF, as a part oi its National Scope Supported Employment Demonstration Project, 
funded by the Office of Special Education and Rcfhabilit ative Services, also has 
conducted a review of exemplary supported employment practices. That review began 
with 183 nominations, including multiple screenings, and resulted in the selecLion 
and on-site verification and review of eight programs with exemplary supported 
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employment practices. From that review of exemplary practices, NARF (1989b) 
identified manager-^ent practices what the eight final sites held in common (Table 3). 



TABLE 3 

Conmon Management Components of Exemplary Sites (NARF, 1989b) 

o Top down commitment to value (a vision followed by mission) 
o Practices that mirror values 

o Networking with others to gain improvement and solve problems 

o An evolving focus on consumers (supported employees) 
including: 

careful monitoring of the effect of the job market 
consumer empowerment and consumer choice 
job satisfaction 

concern for broad quality of life issues 

o Systematic agency self-evaluation of quality through management 
by objectives 



Discussions with the administrators of the eight identified sites over the past 
years as well as discussions with other experts regarding supported employment 
management (Mason, 1990) has led NARF to include a sixth area with the above list, 
o strong organizational development /staff development practices. 

As a part of its review of quality practices, NARF also designed the NARF Quality 
Indicators Profile (NARF, 1989a, Appendix A), That profile was developed to 
evaluate the quality of potential exemplary programs — programs identified first and 
foremost through an emphasis on the outcomes the program achieved for people with 
disabilities. The profile was developed by reviewing the iob satisfaction of 
persons without disabilities and through examination of several instruments designed 
to evaluate the quality of supported employment programs (Allen, Biggs, Sanford, 
Scavarda, & Scott, 1987; Backer, 1986; Bellamy et al,, 1988; CARF, 1990; Mank, 
1988; Schalock, 1988), The instrument which was most influential in the design of 
NARF*s Quality Indicators Profile was developed by Nisbet and Callahan (1989); 
although, for our review purposes we found the format, particular items, additions, 
and differences in orientation significant enough that we developed a separate 
profile. Through that process, the following major categories were identified (Table 
4): 
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TABLE 4 

Categories used to Evaluate Quality of Exemplary Prograns 



o 



o 



o 



o 



o 



Philosophy and Values 
Quality of Life 
Appropriateness of Supports 
Organizational Structure 
Safeguards 



Within the category of Quality of Life (see Table 4) are iteros related to consumer 
choice, work satisfaction, and community empowerment. The Appropriateness of 
Supports contains items related to training services, assessment, and family 
involvement and support. The Organizational Structure section contains items 
concerning organizational stability, management practices, shifting resources, and 
fiscal responsibility. The Safeguards section includes items on preventative 
planning, back-ups for emergencies, and agency self-evaluation/self-monitoring. 

In developing a proposed quality assurance system, NARF reviewed the three 
approaches presented in Tables 2-4: a business approach, categories to evaluate 
quality of exemplary programs, and common management components of exemplary sites. 
The three approaches are remarkably similar, as pernaps they should be since the 
theory upon which NARF built the Quality Indicators Profile came from 1) sound 
business practices and 2) an analysis of components central to quality of life and 
quality of work life for persons without disabilities. 

organizational Davelopnent 

While slightly different terms are used to describe a variety of management and 
staff training functions with the three analyses of quality presented in Tables 2-4, 
the term "organizational development" incorporates most of the traits identified 
through our review. In regard to organizational development, Ivancevich, Szilagyi, 
and Wallace (1977) have proposed a model to assist organizations with rapid change. 
Often the goals of organizational development are improved performance, improved 
motivation, increased cooperation, clearer communications, minimization of conflict, 
and reduced costs. Typically, organizational change involves planned change efforts 
related to the organization's mission. 

It is not our intent to thoroughly cover the field of organizational change, 
however, reference to a few of its tenets may assist individuals desiring to enhance 
the quality of services they provide. The study of organizational development has 
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identified different management styles which may help or hinder the change process. 
The^.f styles are often referred to as: authoritarian, participative, and delegated. 
If administrators are having difficulty with staff retention (as many are) an 
examination of these principles, including a review of matching admini ^trative and 
employee styles, may be useful. 

Another basic concern of the organizational development field is the tempo or pace 
of change. This includes the speed and depth of the process. Organizational change 
theory also describes change as a process which includes the following stages: 
awareness, initial implementation, monitoring, refinement, stabilization, and 
continual monitoring/refinement. 

The change process can be constrained by many variables, including: habits, time, 
money, attitudes, organizational structure or climate, the fear and uncertainty 
surrounding change, individual personality variables, and scarcity of resources or 
conflicting needs. Kurt Lewin's model (Lewin, 1947) to facilitate adjustment to 
change by identifying forces which will positively impact these restraints is a 
cornerstone to overcoming resistance. That model suggests that homeostasis or a 
state of equilibrium is the natural state and that chanyes in one direction will be 
counterbalanced by pulls from the other direction, often negating any impact. To 
overcome the resistance, Lewin recommends carefully targeting specific negative 
forces and designing strategies to shift the equilibrium. 

Ivancevic^i et al. (1977) have identified eleven categories of interventions that may 
be useful to managers. The interventions are centered around activities to diagnose 
problems, enhance employee interactions and skills, develop management capabilities, 
facilitate problem solving and conflict resolution, and further goal-setting and 
planning. While all of the activities appear to have validity for supported 
employment, the greatest efficiency will be achieved through a careful analysis of 
the individual organization. 

Management Needs to Adapt 

A particular concern which many facility administrators face as they implement 
supported employment is related to their own individual lifespace and career 
development. Supported employment administration with an emphasis on 

decentralization of control and the employment of highly autonor^ous staff, varies 
significantly from the traditional rehabilitation model. Supp ;ted employment as 
a concept also varies significantly from the task of many community rehabilitation 
centers, centers which may be directly involved in manufacturing and production 
through a variety of contracts and subcontracts. For a high quality of supported 
employment to be achieved, management must adjust to the differences in 
organizational structure and style which will oest facilitate supported employment 
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in comparison with sheltered activities. Our analysis of the different ways of 
implementing supported employment sugg^srs that there are a variety of paths 
administrators may take; some diagnosis of the best fit between administrative style 
and career path development may assist administrators as they become more involved 
with supported and other integrated community based employment. 

Quality Assurance 

Swedish sociologist Marten Soder (1990) has developed some concepts which also 
should be considered before deciding which quality factors may lead to tne best 
''self-assessment,** Soder *s theory considers the Impact of the passage of time on the 
relevance of the standards being developed. For example, assume that NARF has 
developed quality supported employment standards and that these standards have been 
developed over the course of a year's discussion with experts and consultation with 
the field. Finally, NARF has arrived at the best possible standards and NARF now 
will attempt to disseminate information ana assist the field in adoption and use of 
the standards, [Such a process was used in the development and use of the Quality 
Indicator Profile for supported employment (NARF, 1989a)], 

Soder *s theory is that by the time the field Is ready to use the standards , they are 
outdated. With the qualjty indicator standards developed during 1988 and published 
in 1989, NARF has devoted some efforts in promoting their use, yet in 1990, the use 
is far from widespread. In the mee.ntime, new considerations are evolving with more 
information being obtained concerning most variables, including staff training and 
retention, management competence, costing and funding, and consumer choice. While 
the standards aren*t yet outdated, they may become outdated with major breakthroughs 
that can occur, 

Soder *s theory provided the needed motivation to develop an instrument which was not 
only brief, but one that is designed to "bend with the times" and thus be applicable 
as program standards and needs change, 

NARF's Proposal 

The NARF system includes two basic components: 

1, The NARF Quality Indicators Profile (or related instruments) which can be 
used if you desire a detailed self-checklist that reflects today's values, 
issues and concerns. These may need to be modified as new information is 
acquired, 

2. The six Guiding Principles with an emphasis on "outcomes" first can be a tool 
to guide the organization through evaluation during times of change. The Six 



Guiding Principles are presented in the framework NARF recommends for self 
evaluation in Table 5* 



TABLE 5 



Six Guiding Principles 



Agency 



Date 



Evaluator 



Directions: Rate your agency on a scale of your choosing (we suggest either 
a "high ( + ), medium (0), low or Likert like scale (1-5; 1 = No evidence, 

2 = Needs improvement, 3 = Adequate, 4 = Strong, 5 = Outstanding), 
Mission stateaent: 



Outcomes (wages, hours worked, benefits, integration, level of disability 

served) Criterion : 

Data: 



Strengths: 

Areas Needing Improvement: 
Recommendat ions : 



^Vision (inclusion, participation, eapoweraent, enhanced status, mrk, other) 

Criterion: 
Data: 



Strengths: 

Areas Needing Improvement: 
Recommendat ions : 

Retain objective and criterion Obj. met Ch.inge to: 



Retain objective and criterion 



Ob j . met 



Change to: 



Board Approval 



Board Approval 



Consumer Satisfaction (choice, empowsrment, self-determination) 



Criterion: 



Data: 



Str^;$ngths: 
(cont. on nert page) 
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Areas Needing Improvement: 
RecoiTimendat ions : 

^Retain objective and criterion Obj. met Change to: 

Board Approval 

^Collaboration (planning, funding, implementation, problem solving, 

evaluation) 

Criterion: 
Data: 



Strengths: 

Areas Needing Improvement: 
Recommend at ions : 

^Retain objective and criterion Obj. met Change to: 

Board Approva? 

^Organiiational Development /Staff development (pace of change, management 

style, staff training, differentiated staffing, effective and efficient 
admin i st rat ion ) 

Criterion: 

Data: 



Strengths: 

Areas Needing Improvement: 
Recommend at ions : 

^Retain objective and criterion Obj. met Change to: 

Board Approval 

_Self Evaluation (outcomes, visions, consumer satisfaction, collaboration, 
organiiational development/ staff development) • 

Criterion: 
Data: 



Strengths: 

Areas Needing Improvement: 
Recommendat ions : 

Retain objective and criterion Obj. met rhange to: 

Board Approval 
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Using thm Six Principles 



The six principles can be used by completing the form in Table 5 in somewhat of a 
sequential order. (Agencies may prefer to revise the form so that only cne area is 
evaluated on each page. ) Usually, the evaluator will star.* by completing section one 
on "outcomes" and then revisiting the outcome section after reviewing and revising 
the vision, checking for consistency across sections and practices that mirror 
visions. Agencies should evaluate quality both in terms of the overall organization 
and for the individual. The Six Guiding Principles can be used for individual 
planning to assure quality at the individual level. If agencies opt not to use 
these principles with individuals, then agencies need to consider other ways to 
track individual situations which are not keeping pace with the overall positive 
results obtained for specific time periods. The highest qi^ality will be obtained 
when agencies work on strengthening both overall and individual results. 

An example of a completed form that is ready for board approval is presented in 
Table 6. 



Agency: NARF Affiliate Date June 11, 1990 

Evaluator: C. Mason 

Directions: Rate your agency on a scale of your choosing (we suggert either 
a "high ( + ), nediua (0), low (-)" or Likert like scale (1-5; 1 = No evidence, 
2 = Needs inproyenent, 3 = Adequate, 4 = Strong, 5 = Outstanding). 

Mission statement: 

To assist persons with severe disabilities in securing and maintaining integrated 
community employment. 



_3 OutcoMS (wages, hours worked, benefits, integration, level of disability 

served) Criterion: To place 35 persons in Supported Employment by June 1990. 

Data: 33 placed (6/90) 

Strengths: Types of jobs: industrial, fast food, janitorial, office, vet 
clinic, pet store. Also: 30/33 currently have benefits; 18 are working 30 
hrs/wk, 15 are working 20 hrs or less; 20 diagnosed as mentally ill, 5 
severely MR, 8 moderately MR. 

(cont. on next page) 



TABLE 6 



Six Guiding Principles: An Example 
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Areas Needing Improvement: Need to locate additional placements- 2 short of 
goal. Need to strengthen retention; 8 left jobs during 1989. 
1^ ec ommen da t Ions: 

XJRetaln objective and criterion Obj* met Change to: 

To place 35 by 9/30/90; to maintain 85% of placements. Board Approval 



3 ^Vision (inclusion, participation, empoweraent, enhanced status, work, other) 

Criterion: To collect information on social integration measurement by 9/90 
Data: Identified 4 possible instruments, reviewed with board and staff. 

Strengths: Agreements reached on 10 critical components. 

Areas Needing Improvement: Need to identify system and begin pilot 
measurement . 
/{eco/Tunendations ; 

Retain objective and criterion Obj. met X Change to: 

To select measurement system and begin use by 9/30/90 Board Approval 



3 ^Consumer Satisfaction (choice, enpowerment, self-determination) 

Criterion: To identify possible measures by 6/90 
Data: Identified 3 possible instruments 

Strengths: Have identified instruments for different target populations 

Areas Needing Improvement: Need to implement pilot 
i^eco/Tunendatio/is : 

Retain objective and criterion Ojbj. met^XjOhange to: 

To select instrument and begin pilot use with 3 supported employees by 
9/30/90 Board Approval 



2 Collaboration (planning, funding, implementation, ;^roblem solving, 

evaluation) 

Criterion: To identify 3 possible collaborators and alternative plana for 
collaboration by 6/90 

Data: 3 collaborators identified: MR, MH, residential providers 

Strengths: Initial interest expressed by all three groups 

Areas Needing Improvement: Need to tighten up plans 
Recommen da t Ion s : 

X Retain objective and criterion Ojbj. met Change to: 

To identify 3 possible collaborators and alternative plans 
for collaboration by 9/30/90 Board Approval 



3 Organisational Development/ Staff Development (pace of change, management 

style, staff training, differentiated staffing, affective and efficient 

admini stration) 

Criterion: To conduct 3 staff training activities and identify 2 resources by 
6/90 

Data: Three activities held: job matching, working with TBI, data collection 

Resources identified — examined 5 resources for training activities 
Strengths: Preparing for TBI population; basics covered for new staf^. 



(cont. on next page) 
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Are&a Needing Improvement: Neel a systematic plan to assure training in all 
areas* 

Recommendiitlons: Need to evaluate impact of training and plan& for resource 
use 

Retain objective and criterion OJbj. met X Change to: 

Develop plan for training for 1990-91, incl* resource materials and measure 
impact of training for 1990. Board Approval 



3 Self Evaluation (outcoMS, visions, consumer satisfaction, collaboration, 

organixationr ^ developaant/staf f devalopaent) • 

Criterion : To develop resources and alternatives for expansion into SE 
Data: Have exarined several resources, conducted training, moving in the right 
direction 



Strengths : Have identified feasible paths and costing for each* 

Areas Needing Improvement: Evaluati' ^f impact of training, collaborative 
agreements. 

J?eco/nmendatio/]s: Need to move onto implementation 

Retain objective and criterion Obj* met X Change to: 

To implement pilot studies with evaluation measures and new procedures 

Board Approval 



According tc the example presented in Table 6, the agency viewed its current 
supported employment involvement as adequate in all areas except for collaboration. 
Good progress was generally achieved on the measurable objectives (note the 
specification of specific outcomes and the inclusion of target dates in the 
objectives). The agoncy*s plan was designed for quarterly review due to the rapid 
changes occurring and the need for systematic monitoring. The next step for the 
agency across many of the six areas is "pilot implementation." 

When using the six principles for individual planning, the areas of vision, 
outcomes, and consumer satisfaction are most important. However, for some supported 
employees it will be necessary to develop collaborative efforts and to engage in 
additional organizational development/staff training to meet individual needs. 
Therefore, the areas of collaboration and organization development may sometimes 
need to be addressed from an individual as well as an organizational perspective. 

While this system has its flaws, the basis for the system is the need to define, 
promote, implement, and measure the continual evolution of quality. The management 
by objectives approach is a natural resource for this type of monitoring. 
Additionally, the specification of goals related to outcomos, consumer satisf act: '^n, 
collaboration, and organizational development/staff development should provide 
yardsticks for advancing both supported employment and organizational capacity. 
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Quality a&surance is complex and over the years many complex systems have been 
developed to measure quality* However, starting with a vision that is well-defined 
and that can be stated in realistic, concrete terms has resulted in quality outcomes 
in many business domains and appears to have potential for the non-profit arena as 
well. 

O'Brien (1990) has described the driving force that vision exerts on practices: 

Vision energizes by creating tension with current reality; it communicates 
how the people involved want things to be different* Statements of vision 
feel right and vital to the people concerned, even if they may seem 
strange, impractical, or even foolish to others* Vision can be chosen, but 
cannot be coerced* People do not finish with a vision; rather, as they 
work toward it, their appreciation of its meaning deepens, and the words 
and symbols that communicate it grow richer and clearer. (p 20)* 

Combining vision with information on exemplary practices and an effective self- 
evaluation system should result in quality assurance for supported employment. NARF has 
stated consistently over the past few years that "all rehabilitation practices can 
improve and should improve*'* The format NARF has presented is one vehicle for obtaining 
such improvement. We look forward to the improvements t^ come* 
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